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Abstract 


Fine-grain  concurrent  computers,  by  operating  at  a  fine  grain,  increase  the  amount  of 
concurrency  that  can  be  efficiently  exploited  in  a  given  problem.  Programming  is 
simplified  because  programs  may  be  partitioned  into  natural  units  of  methods  and 
objects  and  these  objects  are  addressed  uniformly  whether  they  are  local  or  remote. 
The  construction  of  these  machines  poses  challenging  problems  in  reducing  overhead, 
increasing  communication  bandwidth,  and  developing  resource  management 
techniques.  This  paper  describes  this  class  of  machines,  the  challenges  posed  by  their 
construction,  and  recent  progress  toward  meeting  these  challenges.  ■<- 


Cr 


"f 


(  / 


.J/ 


88  1122  0-* 


Microsystems 
Research  Center 
Room  39-321 


Massachusetts 
Institute 
of  Technology 


Cambridge 

Massachusetts 

02139 


Telephone 

(617)253-8138 


Acknowledgements 

This  research  was  supported  in  part  by  the  Defense  Advanced  Research  Projects 
Agency  under  contracts  N00014-80-C-0622,  N00014-87-K-0825  and  N00014-85-K-0124, 
and  in  part  by  a  National  Sciences  Foundation  Presidential  Young  Investigators  Award 
with  matching  funds  from  General  Electric  Corporation. 


Author  Information 

Dally:  Department  of  Electrical  Engineering  and  Computer  Science,  Artificial  Intelligence 
Laboratory  and  Laboratory  of  Computer  Science,  MIT,  Cambridge,  MA  02139;  Room 
NE43-41 7,  (61 7)  253-6043. 


Copyright®  1988  MIT.  Memos  in  this  series  are  for  use  inside  MIT  and  are  not 
considered  to  be  published  merely  by  virtue  of  appearing  in  this  series.  This  copy  is  for 
private  circulation  only  and  may  not  be  further  copied  or  distributed,  except  for 
government  purposes,  if  the  paper  acknowledges  U.  S.  Government  sponsorship. 
References  to  this  work  should  be  either  to  the  published  version,  if  any,  or  in  the  form 
“private  communication."  For  information  about  the  ideas  expressed  herein,  contact  the 
author  directly.  For  information  about  this  series,  contact  Microsystems  Research 
Center,  Room  39-321,  MIT,  Cambridge,  MA  02139;  (617)  253-8138. 


Fine-Grain  Message-Passing 
Concurrent  Computers 1 


William  1.  Dally 
Artificial  latslligmr*  liboralory  -* 
Laboratory  far  Computer  Same* 
Maaaackoafats  Iaatitaw  of  Terhantngy 
Cambridge,  Mamschmstt*  02139 


Abstract 


Fin*- grain  concurrent  computer*,  by  operating  at  a  fisc  grain, 
incraaaa  the  amount  of  concurrency  that  can  be  efficiently 
exploited  in  a  given  problem.  Programming  ii  amplified 
became  program*  may  be  partitioned  into  natural  unit!  of 
method*  and  objecta  and  theae  object*  are  addreaeed  uni¬ 
formly  whether  they  are  local  or  remote.  The  coutructioo  of 
theae  machine*  po*e*  challenging  problem*  in  reducing  over¬ 
head,  increasing  communication  bandwidth,  and  developing 
reaource  management  technique*.  Thi*  paper  deacribe*  thi* 
da**  of  machine*,  the  challenge*  poeed  by  their  construction, 
and  recent  progren  toward  meeting  theae  challenge*. 


1  Introduction 

The  frain  fare  of  n  program  refer*  to  the  tize  of  the  talk*  and 
mm* age*  that  make  up  the  program.  Coarm-grain  program* 
have  a  few  long  («  10m*)  task*,  while  fine-grain  program* 
have  many  ahort  (<e  5 p»)  talk*  With  more  talk*  that  can 
execute  at  a  given  time  -  viz.  more  concurrency  -  fine- 
grain  program*  (in  the  absence  of  overhead)  reault  in  fatter 
solutions  than  coarm-grain  program*. 

The  yrnin  *txe  of  a  machine  refers  to  the  physical  size  and  the 
amount  of  memory  in  one  processing  node.  A  coarm-grain 
processing  node  require*  hundreds  of  chip*  (several  boards) 
and  ha*  a:  107  bytes  of  memory  while  fine-grain  node  fit* 

'The  Naaarch  dmribid  i*  tha*  paper  <n*  supported  is  part  by  the 
Mam  Advanced  Raaaarch  Project*  Agency  aider  contract*  N00014- 
•0-C-06K  and  NOOOU-tS-K-OUl  aad  in  part  by  a  NatmaaJ  So¬ 
ane*  fbaadauoa  PnaduMiul  Young  laiwtigaiut  Award  with  matching 
funds  from  General  Electric  Corporation 
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Figure  1:  In  the  area  of  n  1Mbit  DRAM  chip  one  can  con¬ 
struct  n  pmr—ing  node  with  n  32-bit  processor,  n  float¬ 
ing  point  unit,  n  cmnmanicatioo  controller,  and  312Kbit*  of 


on  n  single  chip  and  ha*  *  10*  byta*  of  mtmory.  Fine- 
grain  nodaa  coat  iaaa  and  have  lam  memory  than  coarse- grain 
nodes,  however,  hecaum  ao  little  silicon  area  is  required  to 
build  a  fast  proemme,  they  need  not  have  slower  procemor* 


At  MIT  we  a re  developing  the  J-Machiae  [12]  *a  n  research 
vehicle  to  invastigat*  pcohlama  involved  in  the  design  and 
programming  of  concunaot  computers  with  fine- grain  pro¬ 
emsing  nod**  that  efficiently  execute  fine-grain  programs. 

ProcMorn  am  Intxpsnsiv* 


VLSI  terhnotogy  makm  it  poanMe  to  build  smaU.  powerful 
prn resting  element*.  A  lM-bit  DRAM  chip  ha*  an  area  of 
2S6MA1  (A  is  half  the  minimum  line  width  [23].).  la  the  same 
area  we  can  build  n  single  chip  pro  coming  node  a*  shown  in 
Figure  1.  The  chip  includes 


A  32-bit  prooamor 
A  floating-point  unit 
A  communicatioo  controller 
312Kbits  RAM 


lfiMA’ 

S2MA’ 

SMA1 

128MA* 
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Such  » single- chip  processing  node  would  have  the  net  pro- 
rening  power  u  a  board-sued  node  but  significantly  law 
memory.  Wc  refer  to  a  machine  built  from  tbaee  node*  n  a 
jtllgUm a  m tdunt  ai  it  it  built  with  commodity  part  (jolly- 
baan)  technology. 

A  fine-grain  procoming  nod*  baa  two  major  iilualsgn  den 
(ity  and  memory  bandwidth.  Several  hundred  angle-chip 
nodm  can  be  packaged  on  a  tingle  printed  circuit  beard  per¬ 
mitting  tu  to  exploit  hundreds  of  times  the  concurrency  of 
machines  with  board-used  nodes.  With  on-chip  we 

can  read  an  entire  row  of  memory  ( 128  or  2S6  bite)  in  a  single 
cycle  without  incurring  the  delay  of  several  chip  — 1 ingn 
This  high  memory  bandwidth  allows  the  ■—■"ny  to  simul¬ 
taneously  buffer  mem  ages  from  a  high  bandwidth  network 
and  provide  the  proceseor  with  instructions  and  data. 

Fine  grain  machines  are  quite  efficient.  We  meteure  effi¬ 
ciency  as 

*a  -  1/AT  (1) 

(where  A  is  area  and  T  is  time)  rather  t*>»" 

tN  -  1  /NT  (2) 

(where  IV  is  the  number  of  procesaors).  Proponents  of  coaree- 

grain  machines  argue  that  a  evti—  constructed  from  sev¬ 
eral  thousand  single-chip  nodm  would  be  inefficient 
many  of  the  processing  nodm  trill  be  idle.  N  is  large,  hence 
ejv  is  small.  A  user,  however,  is  not  concerned  with  N,  but 
rather  with  what  the  machine  coats.  A,  and  how  long  it  Ulna 
to  solve  a  problem,  T.  Fine- grain  machines  have  a  very  high 
tA  because  they  are  able  to  exploit  more  concurrency  in  a 
•mailer  area- 


Concurrency  is  Plentiful 

Many  computationally  demanding  problems  have  an  abun¬ 
dance  of  concurrency.  This  concurrency  exists  at  many  lev¬ 
els:  at  the  coarsest  grain  we  iterate  over  the  gridpoints  of 
a  problem.  For  each  gndpoint  we  may  perform  some  vector 
operations  that  can  be  carried  out  in  parallel.  Each  op¬ 
eration  may  involve  the  evaluation  of  some  rxpremions  or 
method  that  can  be  performed  simultaneously.  Within  one 
expression,  several  arithmetic  operations  can  be  performed 
in  parallel. 

At  the  level  of  methods  (subroutines),  the  natural  grain- 
size  of  a  computation  is  10  instructions  [5].  The  message 
transmission  and  reception  overhead  (the  time  for  one  edge  in 
Figure  2)  on  existing  manage  p seeing  computers  is  in  cxcom 
of  500  instruction  times.  As  a  remit  these  "ueiiiM.  operate 
at  a  pain  use  of  2000  instructions.  Conceptually  100  vertices 
of  the  fine-grain  computation  graph  are  grouped  together  to 
amortise  the  communication  and  synchrooiietioe  inirhecrl 
By  reducing  communication  and  synchronisation  overbold  to 
permit  efficient  execution  at  a  grain  site  of  10  instructions 
we  can  exploit  100  times  is  much  concurrency. 


Figure  2:  The  computation  graph  of  a  concurrent  program. 
The  vertices  repremot  n  local  computation  being  pwformsrl 
at  a  node  at  a  concurrent  computer.  The  edges  represent 
oo— mention  actions  betwssn  verticss.  The  time  requited 
to  peefneus  the  computation  is  bounded  below  by  the  tom  of 
edge  and  vertex  than  along  the  critical  path  for  the  compu¬ 
tation. 


A  Global  Addrans  Space  Simplifies  Program¬ 
ming 

A  fine-grain  machine  with  e  global  address  space  simplifies 
programming.  Because  the  machine  executes  programs  at 
their  natural  partition  of  methods  and  objects,  the  problem 
of  partitioning  the  program  into  appropriate  tiaed  pieces  (the 
grouping  of  vertices  in  Figure  2)  is  eliminated.  Each  object  is 
a  separate  partition  and  oech  method  is  separately  scheduled 

A  global  addrns  space  much  of  the  bookkeeping 

required  in  a  system  with  non-uniform  naming.  In  many 
existing  concurrent  computers  local  objects  are  referenced 
through  a  pointer  while  global  objects  require  an  explicit 
and  receive  (30).  Providing  a  global  address  space  al¬ 
lows  objects  to  be  nfuuacod  via  a  single  mechanism  (the 
virtual  address)  regardless  of  their  location,  and  relieves  the 
programmsr  of  tbs  bookkeeping  required  to  keep  track  of 
node  numbers.  Programs  become  both  easier  to  write  and 
more  portable. 


Background 

The  J- Machine  builds  oo  previous  work  in  the  design  of 
mmesge  passing  and  shared  memory  machines.  Like  the  Cal¬ 
tech  Caamic  Cube  (28),  the  Intel  iPSC  (18).  and  the  N-CUBE 
[24],  each  node  of  the  J-Maehiaa  has  n  focal  memory  and 
communicates  with  other  nodes  by  passing  mem ages.  The 
J- Machine  can  exploit  concurrency  at  a  much  finer  grain 
than  them  early  manage  peering  computers.  Delivering  a 
massage  and  dispatching  a  task  in  response  to  the  menage 
arrival  takas  5j*s  oo  the  J- Machine  is  opposed  to  5ms  oo  an 
iPSC.  Like  the  BBN  buttorfiy  {«]  and  the  IBM  RP3  (25)  the 
J-Mnchine  providn  a  global  virtual  addron  space.  The  same 
IDs  (virtual  ad rlrawn)  ere  used  to  reference  oo  and  off  node 
objects.  Like  the  InMOS  transputer  (17)  and  the  Caltech 
MOSAIC  (22)  a  J-Machine  node  is  a  single  chip  processing 


elsmant  integrating  4  proccnor,  memory,  and  a  communi- 
eatioo  unit.  The  J- Machine  extendi  theie  previoui  efforts 
by  providing  efficient  mechanisms  for  supporting  fine-grain 
concurrent  ptogramnung  systems. 

Outline 

The  major  challenge  in  building  a  machine  to  exploit  fine- 
grain  concurrency  is  to  reduce  the  overhead  ssenristnd  with 
massage  sending  and  task  switching  to  a  level  that  is  small 
compared  with  the  task  site.  This  overhead  has  two  com¬ 
ponents,  Tmi,  the  latency  due  to  networks,  and  T— the 
latency  due  to  task  switching  in  a  node.  Low  latency  com¬ 
munication  networks  are  described  in  Section  2.  It  is  shown 
that  low-dimensional  k-ary  n-cube  networks  outperform  bi¬ 
nary  n-cubes  (hypercubes).  To  exploit  the  low-latency  of 
these  networks  requires  processing  elements  that  can  react 
quickly  to  the  arrival  of  massages.  The  architecture  of  such 
a  message-driven  processor  is  described  in  Section  3. 


2  Interconnection  Networks 

VLSI  systems  are  wire  limited.  The  cost  of  these  systems 
is  predominantly  that  of  connecting  devices,  and  the  per¬ 
formance  is  limited  by  the  delay  of  these  interconnections 
Thus,  an  interconnection  network  must  make  efficient  use  of 
the  available  wire.  The  topology  of  the  network  must  map 
into  the  three  physical  dimensions  so  that  messages  are  not 
required  to  dosifr  keck  on  themselves,  and  in  a  way  that 
allows  messages  to  use  all  of  the  available  bandwidth  along 
their  path.  Also,  the  topology  and  routing  algorithm  mast 
be  simple  so  the  network  switches  will  be  sufficiently  fast  to 
avoid  leaving  the  wires  idle  while  making  routing  decisicns. 

Our  recent  findings  suggest  that  low-dimensional  k-ary  n- 
cube  interconnection  networks  [7]  using  wormhole  renting 
(27)  [19]  and  virtual  channel*  [8]  are  capable  of  providing  the 
performance  required  by  fine-grain  concurrent  architectures. 
To  test  these  ideas,  we  have  constructed  two  prototype  VLSI 
routing  chips,  the  torus  routing  chip  (TRC)  [6],  and  the  net¬ 
work  design  frame  (NDF)  [10]  The  mesh  routing  chip  MRC 
[14],  based  on  similar  principles,  has  been  applied  in  a  com¬ 
mercial  product  [2]. 

Wormhole  Routing 

With  wormhole  mutiny  (Figure  3B)  as  soon  as  each  flit  (flow- 
control  digit)  of  a  message  arrives  at  a  node  it  is  forwarded  to 
the  next  node.  With  itere-and-forward  mating  (Figure  3A), 
the  method  used  by  most  existing  concurrent  computers,  the 
entire  mem  age  it  received  before  forwarding  the  packet  to  the 
next  node.  Using  wormhole  routing  gives  a  network  latency, 
Twh,  that  is  the  sum  of  a  component  due  to  message  length 
normalised  to  channel  width  £,  and  a  component  due  to 
the  distance  the  message  must  travel.  D  With  stsm-end- 
jorwari  routing,  on  the  other  hand,  the  latency,  Tsr ,  is  the 


Figure  3:  The  latency  of  stem  sad  forward  resting  ( A )  com¬ 
pared  to  wsrmksfs  routing  (B)  Wormhole  routing  reduces 
latency  from  the  product  of  and  D  to  the  sum  of  these 
two  components. 


product  of  these  two  components. 

Tw»-Tc(^  +  d).  (3) 

Ttr  ■  Tc  (jjr  *  ,  (4) 

where  To  is  the  channel  transmission  time,  I  is  the  message 
length  in  bite,  W  is  the  channel  width  in  bits,  and  D  is  the 
number  of  rh annals  the  message  must  traverse  (distance). 

Consider  a  concurrent  computer  with  64K  nodes  connected 
as  a  16-ary  4-cube  with  6-bit  wide  channels  ( W  m  8).  Assum¬ 
ing  no  locality,  the  average  distance  a  message  must  travel  in 
this  machine  is  f)  w  15.  For  2S6-bit  massages,  Twh  m  477c, 
an  order  of  magnitude  lass  than  T$r  “  4807c 


Low-Dim*  ns  io  nil  k-ary  n-Cubes 

Many  concurrent  computers  have  been  built  using  binary 
n-cube  (hyperenbe)  interconnection  networks  because  these 
networks  an  optimal  when  all  channels  are  considered  equal 
However,  considering  a  channel  in  a  binary  n-cube  to  be 
equal  to  a  channel  in  a  low -dimensional  network  it  not  s 
reasonable  mnmption.  Because  binary  n-cubes  have  long 
wins  and  high  bisection  widths  their  channels  are  typically 
narrower  end  slower  then  the  channels  in  a  low-dimenaional 
network.  When  these  factors  ate  taken  into  account,  the 
low -dimensional  networks  out-perform  the  high-dimensional 
networks. 

Consider  the  oet works  shown  in  Figure  4.  Suppose  the  bi¬ 
nary  6-cube  has  4-bit  wide  channels  (ss  in  the  Caltech  Cos¬ 
mic  Cube  (38)).  An  8-ary  2 -cubs  srith  16-bit  wide  channels 
has  the  tame  wiring  complexity.  With  wormhole  routing  and 
286-bit  mamagee  the  6-cube  has  a  latency  of  67Tc  while  the 
2-cube  has  a  latency  of  only  207c-  Increasing  the  radix,  k. 
of  a  k-ary  n-cube  while  holding  wiring  complexity  (bisection 
sridth)  constant  increases  both  W  at  k  and  D  oc  kn.  This  de- 
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creates  the  coapooent  of  latency  due  to  mmip  length,  (j), 
while  increasing  the  component  due  to  distance,  D  The  min¬ 
imum  latency  occurs  when  theee  two  component*  an  nearly 
equal  (Figure  3).  For  £  *  200  the  optimum  dimension,  n,  it 


Figure  4:  Two  64-node  4-ary  n-enbee:  an  6-ary  2 -cube  (A) 
and  a  binary  6-cube  (B).  Network  A  has  a  bisection  width 
of  16  while  B  ha*  a  bisection  width  of  64  channels. 

Thus  the  channel*  in  A  can  be  made  four  times  tt  wide  at 
the  channels  in  B  for  the  same  wiring  complexity. 


Fieure  S:  Latency  as  a  function  of  dimension  for  networks  of 
constant  bisection  width  (B»N,  L»160).  Low-dimensional 
network*  (left)  are  distance  limited,  while  higb-dimensional 
networks  (right)  are  message- length  limited. 


Figure  6:  Latency  vs.  Draffic  foe  a  32-ary  2-cube.  Lw200bits. 
Solid  Una  is  predicted  latency,  paint*  ate  masauramsnt*  taken 


(1024  nodes).  The  solid  line  is  the  predicted  latency.  The 
points  era  maasereoanta  takan  from  a  simulator.  The  model 
ay  ess  with  the  simulation  within  891,  with  the  model  being 
slightly  psssitniatiri  until  the  network  approaches  saturation. 
Latency  incenses*  lass  than  20%  a*  tragic  is  increased  from 
sen  to  30%  capacity.  Saturation  (maximum  throughput) 
occurs  at  «  40%  capacity. 


a  B arena*  wire*  ate  shorter,  the  channel*  in  thane  net¬ 
works  typically  operate  faster  than  in  high  dimensional 
networks,  ionesoing  throughput  and  further  decreas¬ 
ing  latency. 

a  low  dimensional  uatwoska  have  better  queueing  per¬ 
formance.  If  oot  thinks  of  channels  as  being  servers, 
these  nstmoeka  have  ftwur  server*  with  greater  capacity 
nonlting  in  a  lower  average  service  time. 

a  Because  the  cotitroi  logic  for  a  network  switch  typically 
scales  with  the  number  of  dimensions,  the  switches  for 
low-dimansional  networks  are  simpler  than  thoee  for 
high-dinwnaional  network*. 


two  for  up  to  IK  node*  and  three  for  IK  to  32K  nodes,  end 
four  for  32K  to  1M  nodes 

The  throughput  of  a  network  is  the  maximum  number  of 
imnssgrs  that  can  be  delivered  per  unit  time.  It  is  often  ex¬ 
pressed  a*  a  fraction  of  the  network’s  capacity,  the  number 
of  massages  that  would  be  delivered  if  every  channel  of  the 
network  was  fully  used.  As  the  amount  of  traffic  in  the  net¬ 
work  increases,  the  latency  of  a  message  is  increased.  The 
latency  given  by  (3)  assumes  an  unloaded  network. 

We  have  developed  a  queueing  model  of  k-ary  n-cube  worm- 
bole  networks  that  accurately  predict*  the  latency  a*  a  func¬ 
tion  of  network  traffic,  and  allow*  us  to  calculate 

throughput  for  a  given  network  configuration  (7).  Fig¬ 
ure  6  show*  bow  latency  varies  with  traffic  for  a  32-ary  2-eube 


Virtual  Channels 

Until  recently  there  was  no  known  algorithm  for  deadlock- 
free  routing  is  k-ary  w-eebe,  wormhole  networks.  The  con¬ 
ventional  sriedersd  kmfftr  peof  algorithms  that  are  used  in 
store- and-forward  network*  are  oot  applicable  to  networks 
that  uaa  wormhole  renting.  Three  algorithms  interleave  the 
items  being  buffered  (packet*  in  a  store- and-forward  net¬ 
work),  but  wormhole  networks  buffer  flit*  that  cannot  be 
interleaved. 

We  have  developed  a  new  dare  of  algorithms  for  deadlock 
free  ranting  breed  on  the  concept  of  wrteef  chmnmit  Shown 
in  Figure  7,  virtual  channel  algorithms  operate  by  restrict- 


«l 


Figure  7:  Considering  routing  to  be  s  function  C  x  N  >-*  C 
rather  than  the  conventional  N  x  N  <-»  C  deadlock  corre¬ 
sponds  to  cycles  in  the  channel  dependency  graph  (right) 
rather  than  the  interconnection  graph  (left).  By  multiplex¬ 
ing  two  virtual  channels  on  each  physical  channel,  we  can 
restrict  the  touting  function  to  eliminate  deadlock  (bottom). 


ing  routing  rather  than  by  restricting  buffer  allocation.  To 
do  this  requires  that  routing  be  a  function  of  tbe  channel  a 
message  arrives  on  and  the  destination  node,  C  x  N  •—  C, 
rather  than  the  node  a  mesa  age  is  on  and  the  destination 
node,  JV  x  yv  >->  C.  Projecting  this  function  gives  a  de¬ 
pendency  relation  among  channels.  By  multiplexing  several 
virtual  channels  on  each  physical  channel  we  can  restrict 
routing  in  a  manner  that  avoids  deadlock  without  loosing 
strong  connectivity.  A  set  of  virtual  channels  all  share  the 
same  physical  wires.  Each  virtual  channel  requires  only  a 
single  flit  buffer.  The  virtual  channel  method  can  be  used  to 
route  deadlock  free  in  any  strongly  connected  network  [8]. 

The  Torun  Routing  Chip 

The  Torus  Routing  Chip  (TRC),  shown  in  Figure  8,  is  a 
self-timed  [26]  VLSI  chip  that  performs  wormhole  routing  in 
k-ary  n-cube  networks,  and  uses  virtual  channels  to  prevent 
deadlock  [6].  A  single  TRC  provides  8-bit  data  channels  in 
two  dimensions  and  can  be  cascaded  to  add  more  dimensions. 
A  TRC  network  can  deliver  a  190-bit  message  in  a  1024  node 
32-ary  2-cube  with  an  average  latency  of  7.5 pa. 

The  Network  Design  Frame 

The  Network  Design  Frame  (NDF)  [10]  incorporates  a  parti¬ 
tioned  switch  architecture  [14],  bidirectional  data  channels, 
and  low-voltage  output  drivers  to  achieve  a  worst-case  la¬ 
tency  of  3ps  in  a  4K  node  64-ary  2-cube.  In  tbe  partitioned 


switch  architecture,  shown  in  Figure  8,  the  routing  logic  is 
partitioned  into  two-way  switches.  The  partitioned  switch's 
data  paths  and  control  logic  are  simp  1st  (and  thus  smaller 
and  (aater)  than  the  ceutraliaod  crossbar  design  used  in  tbe 
TRC.  A  signal  rooms  through  only  10  gate  delays  horn  input 
to  output  (or  o  propagation  delay  of  20ns  (estimated). 

Bidirectional  data  channels  an  used  in  the  NDF  to  reduce 
latency  and  to  exploit  locality.  Because  wire  density  is  a 
mhjnff  UmiUUoc,  tbe  two  of  iwwinnm[f»t  will 

•hare  the  same  data  wires.  While  the  NDF  is  constructed  us¬ 
ing  CMOS  technology,  cnanmemraiioa  on  tbeee  bidirectional 
data  wins  nets  ECL  hgnal  levels  to  improve  speed,  reduce 
power  dish  police,  and  reduce  noise.  The  NDF  uses  low- 
voltags  swing  output  pads  boned  oo  adsstgn  by  Knight  |20J. 
Reducing  tbe  voltage  swing  by  a  factor  of  9  makes  these  pads 
S  times  os  (sot  as  coevantimial  pads.  Also,  because  power 
goes  m  the  square  of  voltage,  P  m  CV*J ,  thaw  pad*  dissi¬ 
pate  1/29  (4%)  as  much  power  se  conventional  pads.  Since 
much  of  the  power  in  the  goes  into  driving  the  inter- 

node  wine,  this  aavinp  rspcassnta  a  considerable  redaction 
in  total  power  dissipation. 


Adaptive  Routing 

The  TRC  and  NDF  are  oblivious  routers  -  via.  the  route 
selected  for  a  msassge  it  determined  only  by  tbe  source  and 
dmtination  nodes.  In  particular,  they  route  a  mesaage  first 
in  tbe  X  direction  and  then  in  the  Y  direction.  Aa  shown 
in  Figure  ID  if  several  sources  having  the  same  Y  coordinate 
transmit  trass  sgaa  to  several  destinatkms  having  the  same  X 
coordinate  only  one  mem  age  can  proceed  at  a  time1. 

As  shown  in  Figure  11,  simply  relaxing  the  X-Y  routing  or 
der  could  result  in  deadlock.  The  deadlock  can  be  avoided  by 
the  virtual  than — *•  in  tbe  north  and  south  direc¬ 
tions  to  separate  east  bound  massages  from  westbound  mes¬ 
sages  [21].  We  have  recently  undertaken  the  design  of  an 
adaptive  router  chip  (ARC)  baaed  on  this  technique. 


3  A  Message-Driven  Processor 

Conventional  instruction  preceMoes  a re  ill-suited  to  serve  as 
pro  reusing  nodes  in  a  concurrent  computer.  Their  I/O  sys¬ 
tems  an  designed  to  handle  high-latency  peripherals  (e.g., 
disks)  and  thus  they  respond  slowly  (m  100  instruction  times) 
to  maaaigai  arriving  over  the  network.  Also,  their  register- 
oriented  instruction  sets,  designed  to  match  a  fast  processor 
with  a  alow  tnsmocy  in  programming  environments  where 
context  switches  am  infrequent  (1  in  w  29000  instructions), 
are  not  appropriate  in  a  processing  node  containing  a  fast 
local  memory  and  in  an  environment  where  context  switches 
happen  every  10  instruct  ions. 


'Only  set  of  tbs  two  eanditiees  (seuret 
X  coordinates)  mast  be  pevaeat  to  cause  o 
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Figure  11:  (A)  Relaxing  the  X-Y  routing  order  retuJu  in 
cycle*  in  the  channel  dependency  graph  and  thu*  a  potential 
deadlock.  (B)  To  prevent  deadlock  we  can  add  additional 
virtual  channels  to  separate  eaatbound  messages  from  west¬ 
bound  messages. 


The  solution  adopted  in  many  machines  is  to  increase  the 
memory  size  of  the  node  so  a  larger  part  of  the  problem  can 
be  performed  in  each  node.  This  baa  the  effect  of  reducing 
the  concurrency  to  a  point  where  the  number  of  instructions 
executed  between  messages  exceeds  10*.  This  increases  the 
perceived  efficiency  from  10%  to  <0%  when  measured  in 
terms  of  tpi  (2).  This  measure  of  efficiency,  however,  ignores 
the  cost  of  the  node.  If  instead  we  measure  efficiency  in 
terms  of  tA  (1),  the  actual  efficiency  has  been  reduced  by 
making  the  node  larger.  To  truly  increase  the  efficiency,  we 
must  build  small,  efficient  node*. 

At  MIT,  we  are  developing  the  mins  age-driven  processor 
(MDP),  a  small,  efficient  processing  node  for  a  massage 
passing  concurrent  computer  (9).  It  is  designed  to  support 
fine-grain  concurrent  programs  by  reducing  the  overhead  and 
latency  associated  with  receiving  a  message,  by  reducing  the 
time  necessary  to  perform  a  context  switch,  and  by  providing 
hardware  support  for  object-oriented  concurrent  program¬ 
ming  systems. 

The  MDP  provides  the  following  mechanisms 

1.  A  tend  instruction  to  inject  short  massages  into  the 
network  with  a  minimum  of  delay. 

2.  A  message  unit  that  controls  the  reception  and  buffer¬ 
ing  of  messages. 

3.  A  scheduling  mechanism  that  decide*  when  to  preempt 
execution  and  selects  a  message  to  be  executed  when 
a  method  suspends. 

4.  A  general  translation  mechanism. 

5.  A  small  processor  state  and  two  sets  of  processor  reg¬ 
isters  to  support  fast  task  switches. 


The  MDP  injects  message*  into  the  network  using  a  tend 
instruction  that  transmit*  ooe  or  two  words  (at  moat  one 
bom  memory)  and  optionally  terminates  the  message.  The 
first  word  of  the  massage  is  interpreted  by  the  network  a* 
an  absolute  nods  address  (in  xjr  format)  and  is  stripped  off 
before  delivery.  The  remainder  of  the  msssigf  ia  transmit¬ 
ted  without  modification.  A  typical  mssaags  sand  is  shown 
in  Figure  12.  The  first  instruction  sends  the  absolute  ad¬ 
dress  of  the  destination  nod*  (contained  ia  RO).  The  second 
instruction  sends  two  words  of  data  (bom  hi  and  R2).  The 
final  instruction  sends  two  additional  words  of  data,  one  from 
*3,  and  on*  from  msmory.  The  use  of  th*  SUDE  instruction 
marks  the  end  of  the  mssaags  and  causes  it  to  be  transmit¬ 
ted  into  the  network.  In  n  Concurrent  Smalltalk  message, 
the  first  word  it  a  mmtsgs  header,  the  ssooad  specifies  the 
receiver,  the  third  word  is  the  selector,  subsequent  words 
contain  argument*,  and  the  final  weed  ia  a  continuation.  On 
our  register- transfer  simulator,  this  sequence  executes  in  4 
dock  eyelet. 

Early  in  the  design  of  the  MDP  we  considered  making  a  mes¬ 
sage  sand  n  singis  instruction  that  took  a  msmage  template, 
filled  in  the  template  using  the  current  addressing  environ¬ 
ment,  and  transmitted  the  message.  Each  template  entry 
specified  on*  word  of  the  message  aa  being  either  a  constant, 
the  contents  of  a  data  register,  or  a  memory  reference  offset 
from  aa  address  register  (like  an  operand  descriptor).  The 
template  approach  was  abandoned  in  favor  of  the  simpler 


SOn>  RO  ;  sand  net  address 

SESD2  R1.R2  ;  header  end  receiver 

SETOSE  M, selector  and  continuation  -  and  nag. 


Figure  12:  MDP  assembly  code  to  send  a  4  word  massage 
use*  three  variants  of  the  SOT)  instruction 


ooe  or  two  operand  SOT)  instruction  because  the  template 
did  not  significantly  reduce  code  space  or  execution  time.  A 
two  operand  SOT)  instruction  results  in  code  that  is  nearly  as 
dense  as  a  template  end  can  be  implemented  using  the  same 
central  logic  used  for  arithmetic  and  logical  instruction*. 

Massage  Reception 

Massage  reception  overhead  ia  reduced  to  *  lp*  by  buffer¬ 
ing,  scheduling,  and  dispatching  messages  in  hardware.  The 
MDP  maintains  two  mi  age/scheduling  queues  (correspond¬ 
ing  to  two  priority  levels)  in  it*  on-chip  memory.  As  massage* 
arrive  over  the  network,  they  are  buffered  in  the  appropriate 
queue.  The  queues  are  implemented  as  circular  buffers.  It  is 
important  that  the  queue  have  sufficient  performance  to  ac¬ 
cept  words  from  the  network  at  the  tame  rate  at  which  they 
arrive.  Otherwise,  messages  would  backup  into  the  network 


« 
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causing  congestion.  To  achieve  the  required  performance, 
special  addressing  hardware  it  uted  to  enqueue  or  dequeue 
a  mamage  word  with  wraparound  and  full/empty  check  in 
a  tingle  clock  cycle.  A  queue  row  buffer  allows  enqueuing 
to  proceed  using  one  memory  cycle  for  each  four  wordt  re¬ 
ceived.  Thua  a  program  can  execute  in  parallel  with  menage 
reception  with  little  lorn  of  memory  bandwidth. 

The  MDP  ached ulet  the  task  taeocitted  with  each  queued 
manage.  At  any  point  in  time,  the  MDP  m  executing  the 
taak  aaaociated  with  the  firet  manage  in  the  higheat  prior¬ 
ity  non-empty  queue.  If  both  queun  are  empty,  the  MDP 
it  idle  -  via.,  executing  a  background  taak.  Sending  a  mee- 
tage  implicitly  scbeduln  a  taak  on  the  deatinalioo  node.  The 
taak  will  be  run  when  it  roacbn  the  head  of  the  queue.  This 
timple  two-priority  tcbeduling  mochaniam  removn  the  over¬ 
head  taaocialed  with  a  toftware  tcheduler.  More  aophiati- 
ceted  tcbeduling  policin  may  be  implemented  on  top  of  tbit 
lubatrate. 

Meaaagn  become  ecfiec  either  by  arriving  while  the  node  it 
idle  or  executing  at  a  lower  priority,  or  by  being  at  the  head 
of  a  queue  when  the  preceding  menage  tutptwdt  execution. 
When  a  menage  bocomn  active,  a  handler  it  ditpatchad  in 
ooe  clock  cycle.  The  dispatch  fortea  execution  to  a  phyncal 
addren  specified  in  the  manage  header.  Thit  mechanism  it 
uted  directly  to  procan  mantgn  requiring  low  latency  (e.g., 
combining  and  forwarding).  Other  mntagn  (e.g.,  remote 
procedure  call)  tpecify  a  handler  that  locetn  the  requited 
method  (using  the  tranalatioo  merhanitm  described  below) 
and  then  transfers  control  to  it. 

For  example,  the  call  handler  code  is  shows  in  Figure  13  and 
its  execution  ia  depicted  m  Figure  M.  The  first  instruction 
gets  the  method  ID  (offset  1  into  the  message).  To  facili¬ 
tate  access  to  the  menage  arguments,  hardware  initialises 
register  A3  to  contain  an  addren  descriptor  (ban/length) 
for  the  current  menage.  The  next  instruction  translates  the 
method  ID  into  an  addren  descriptor  for  the  method.  If 
the  translate  (suits,  because  the  method  is  not  resides!  or 
the  descriptor  is  not  in  the  cache,  the  fault  handler  fitu 
the  problem  and  reschedules  the  menage.  If  the  translation 
succeeds,  the  final  instruction  (resume)  transfers  control  to 
the  method.  The  method  code  may  then  read  in  arguments 
from  the  message  queue.  The  argument  object  identifiers  are 
translated  to  physical  memory  baee/length  pairs  using  the 
translate  instruction.  If  the  method  needs  space  to  store  lo¬ 
cal  state,  it  may  create  a  context  object.  When  the  method 
has  finished  execution,  or  when  it  needs  to  wait  for  a  reply, 
it  executes  a  SUSPEND  instruction  paning  control  to  the  next 
message. 


HOVE  Cl.A3l.A0;  get  nethod  id 

ELATE  HO, AO  ;  translate  to  address  descriptor 

US  3  :  transfer  control  to  nathod 

Figure  13:  MDP  assembly  code  for  the  CA1X  maw  age. 


Figure  14:  The  CALL  mamage  invokes  a  method  by  translat¬ 
ing  tha  method  identifier  to  find  the  coda,  creating  a  context 
(if  anomaary)  to  hold  local  stale,  and  translating  argument 
identifier!  to  locate  arguments. 


An  early  vnmon  of  tha  MDP  had  a  fixed  set  of  truss  age  han¬ 
dlers  in  macrocode.  An  analysis  of  tbaae  handlers  showed 
that  their  performance  win  limited  by  memory  ate— as.  Thus 
than  was  little  advantage  in  using  microcode  The  mi¬ 
crocode  was  eliminated  tha  handlers  were  recoded  in  as¬ 
sembly  language,  and  tha  message  speed*  was  defined  to  be 
the  physical  address  of  the  handler  routine.  Frequently  used 
handlers  an  rent  lined  ia  an  on-chip  ROM.  This  approach 
“t—pTC***  tha  central  structure  of  the  machine  and  gives  us 
fiexibility  to  rtddbe  message  handlers  to  fix  bugs,  for  in¬ 
strumentation  (o-g.,  to  count  tha  number  of  sands),  and  to 

The  mmsage  queue  originally  sllocaUd  storage  from  the  heap 
far  each  incoming  nreengi.  This  eliminated  the  need  to  copy 
meeeiges  what  a  method  suspended  foe  intermediate  raeults. 
However,  tha  cost  of  allocating  and  reclaiming  storage  for 
each  mamage  proved  to  be  prohibitive  Instead,  we  settled 
on  the  praellocated  circular  buffer.  When  a  method  suspends 
for  intermediate  raeults,  mamage  arguments  are  copied  into 
a  context  object.  The  overhead  of  this  copying  is  small  since 
the  oontext  must  be  creeled  anyway  to  specify  a  continuation 
and  to  hold  lire  variables.  The  fixed  buffer  also  provides  a 
convenient  layering.  Priority  aero  messages  are  sect  when  tbe 
memory  s’facator  runs  out  of  room  and  priority  ooe  mem igee 
are  cant  when  the  priority  Mto  queue  fills. 


TfanaUtion 

Tbe  MDP  ia  an  experiment  ia  unifying  shared-memory  and 
mmatge  peering  parallel  computers.  Shared-memory  ma¬ 
chines  provide  a  uniform  global  name  space  (address  space) 


that  allows  processing  dementi  to  irrm  date  rcg ttdlem  of 
iti  location.  Ilemato  fusing  «e»chi»*.  perform  CQBUBmUC4» 
tioa  and  synchronization  via  oode-to-node  messages.  Them 
two  concept*  am  not  mutually  exclusive.  The  MDP  providca 
a  virtual  ■ddramiaf  mechaniim  intended  to  tupport  a  flobal 
name  rpace  while  uiinf  as  execution  mechaniim  bawd  on 
meraage  paming. 

The  MDP  implement!  a  global  virtual  addtem  (pace  uiing 
a  very  general  traadatioo  mechanical.  The  MDP  memory 
allowi  both  indexed  and  mt-amodative  accam.  By  build¬ 
ing  comparatori  into  the  column  multiplexer  of  the  on-chip 
RAM,  we  are  able  to  provide  aet-aiaociative  accam  with  only 
a  small  incruaein  the  fixe  of  the  RAM 's  peripheral  circuitry. 

The  traadatioo  mechaniim  ii  expoeed  to  the  programmer 
with  the  arm  and  XLATE  iaatructiooi.  OTTO  Ra.Ab  amo- 
ciatea  the  contenu  of  Ra  (the  key)  with  the  coo  ten  u  of  Rb 
( the  data).  The  amociatioo  ii  made  on  the  full  36  bit!  of  the 
key  10  that  tag!  may  be  uaed  to  diatinguilh  different  keyi 
XLATE  Ra.Ab  look!  up  the  data  amodated  with  the  content! 
of  Ra  and  itorei  thii  data  in  ah.  The  instruction  fault!  if 
the  lookup  mime!  or  if  the  data  is  not  an  address  descrip¬ 
tor.  XLATE  Ra.Ab  can  be  used  to  lookup  other  types  of  data. 
This  mechanism  is  used  by  our  system  code  to  cache  ID  to 
addtem  descriptor  (virtual  to  physical)  translations,  to  cache 
ID  to  node  number  (virtud  to  physied)  translations,  and  to 
cache  class/selector  to  addtem  descriptor  (method  lookup) 
translations. 

Tags  are  an  integrd  part  of  our  addressing  mechanism.  An 
ID  may  translate  into  an  ad  dram  descriptor  for  a  local  object, 
or  a  node  addtem  for  a  global  object.  The  tag  allows  us  to 
distinguish  these  two  cases  and  a  fault  provides  an  efficient 
mechanism  for  the  test.  Tags  also  allow  us  to  distinguish  an 
ID  key  horn  a  clam /selector  key  with  the  same  bit  pattern. 

Most  computers  provide  a  set  associativa  cache  to  accelerate 
translations.  We  have  taken  this  mechanism  and  expoeed 
it  in  a  pair  of  instructions  that  a  systems  programmer  can 
use  for  any  translation.  Providing  this  general  mechanism 
gives  us  the  freedom  to  experiment  with  different  address 
translation  mechanisms  and  different  uses  of  translation.  We 
pay  very  little  for  this  flexibility  since  performance  is  limited 
by  the  number  of  memory  accesses  that  must  be  performed. 

Context  Switches 

Context  switch  time  is  reduced  by  making  the  MDP  a  mem¬ 
ory  rather  than  register  baaed  processor.  Each  MDP  in¬ 
struction  may  read  or  srrite  one  word  of  memory.  Because 
the  MDP  memory  is  on-chip,  these  memory  references  do 
not  slow  down  instruction  execution.  Four  general  purpoee 
registers  are  provided  to  allow  instructions  that  require  up 
to  three  operands  to  execute  in  a  single  cycle.  The  entire 
state  of  a  context  may  be  saved  end  restored  in  lees  than 
12  dock  cydes.  Tsso  register  sets  are  provided,  one  for  each 
of  two  priority  levels,  to  allow  low  priority  messages  to  be 
preempted  without  saving  state. 


Synchronisation  using  Ifcgs 

An  MDP  word  is  36-bits:  a  4-bit  tag  and  a  32-bit  datum. 
Tags  ase  wed  both  to  support  dynamically-typed  program¬ 
ming  languages  and  to  tupport  concurrent  programming  con¬ 
structs  such  so  relocatable  object!  and  futures. 

For  example,  consider  the  case  where  an  object.  A,  sends 
n  message  to  an  object,  B,  instructing  B  to  perform  some 
computation  and  than  to  return  the  result  in  a  reply  message 
to  updata  A's  local  variable  *.  To  synchronise  with  the  reply, 
A,  first  tags  t  aa  a  C-FBT  (for  context  future)  then  sends 
the  memsge  and  procoode  without  waiting  far  a  reply.  If  the 
reply  arrive*  before  A  nets  *  execution  simply  continues.  An 
attempt  to  use  *  before  the  reply,  however,  results  in  a  trap 
that  suspends  execution  until  the  reply  arrives. 

The  Effects  of  a  Small  Memory 

Because  the  MDP  maintains  a  global  name  space,  it  is  not 
necessary  to  koap  a  copy  of  tbs  program  code  (aad  the  op¬ 
erating  system  code)  at  each  node.  In  fact,  a  copy  of  the 
entire  operating  system  will  not  fit  into  e  node's  memory. 
Each  MDP  keeps  a  method  cache  in  its  memory  and  fetches 
ipeib"*1*  from  a  single  distributed  copy  of  the  program  on 
cache  mi  si  as. 

Some  may  argue  that  the  MDP  is  unbalanced  according  to 
the  rule  of  thumb  stating  that  a  1MIP  processor  should  have 
a  1MByte  memory.  The  MDP  is  an  w  «MIP  processor  and 
only  has  a  36KByte  memory.  We  argue  however  that  it  is 
not  the  sue  of  the  memory  in  a  single  node  that  is  important, 
but  rather  the  amount  of  memory  that  can  be  secerned  in 
a  given  period  of  time.  In  a  64K  node  machine  constructed 
from  MDPt  aad  using  a  fast  routing  network,  a  processor 
will  be  able  to  access  a  uniform  ad  dree*  space  of  2*  words 
(2"  Bytes)  in  Isas  than  10ps. 

The  MDP  provides  many  of  the  advantages  of  both  message- 
pmeing  multi  computers  and  sharad-memory  multiprocessors. 
Like  a  shared-memory  machine,  it  provides  a  single  global 
name  space,  and  needs  to  keep  only  s  single  copy  of  the  ap¬ 
plication  aad  operating  system  code.  Like  a  message- passing 
machine,  the  MDP  exploits  locality  in  object  placement,  uses 
■Damages  to  trigger  events,  and  gains  efficiency  by  sending  a 
single  mamags  through  the  network  instead  of  sending  mul¬ 
tiple  words.  While  we  plan  to  implement  no  object-oriented 
programming  system  on  the  MDP,  we  also  tee  the  MDP  ss 
an  smulnior  that  can  be  used  to  experiment  with  other  pro- 
gruo2&io|  owdili. 


4  Conclusion 

The  J-Machine  efficiently  executes  fine-grain  concurrent  pro¬ 
grams  by  providing  mechanisms  that  reduce  the  overhead 
of  meossge  passing,  translation,  aad  context  switching  to 
ns  Jjss.  Reducing  overhead  to  a  time  comparable  with  the 
natural  grain  site  of  many  concurrent  programs  allows  the 


programmer  to  exploit  all  of  the  concurrency  ptcaent  in  theae 
programs  rather  than  grouping  many  grains  together  -  re¬ 
ducing  the  concurrency  to  improve  the  efficiency. 

Loo-dimensional  t-ary  n-cube  networks  that  use  wormhole 
routing  and  virtual  channels  can  send  a  6-word  message 
across  the  diameter  of  a  4K-node  concurrent  computer  in 
4«is.  These  low-dimensional  networks  (8  <  *  <  64  and 
2  <  n  <  4)  outperform  binary  n -cubes  ( k  «  2)  because 
they  balance  the  component  of  latency  due  to  message  length 
with  the  component  due  to  distance.  These  networks  are  im¬ 
plemented  with  VLSI  chips  such  as  the  TRC  [6],  the  NDF 
[10),  and  the  MRC  [14]  that  perform  all  routing  and  buffer¬ 
ing  internally  using  no  memory  bandwidth  or  CPU  time  on 
intermediate  nodes.  Adaptive  routers  are  being  developed 
that  will  further  improve  routing  performance  by  reducing 
contention. 

The  Message-Driven  Processor  (MDP)  can  perform  a  task 
•witch  on  message  arrival  in  lpa.  The  MDP  performs  mes¬ 
sage  reception,  buffering,  and  scheduling  in  hardware  to  elim¬ 
inate  the  software  overhead  of  lOOus  or  mote  amodated  with 
these  functions.  Task  switches  we  performed  quickly  because 
the  MDP  is  memory  rather  than  register  based.  The  MDP 
memory  provides  both  associative  and  indexed  access.  The 
associative  access  is  used  to  support  a  global  virtual  address 
spare  needed  to  support  concurrent  programming  systems. 
The  MDP  provides  very  general  hardware  mechanisms  that 
can  support  many  different  concurrent  programming  mod¬ 
els  including  conventional  message  patting  [30],  acton  (1) 

[3].  futures  (1SJ,  communicating  processes  [16],  and  dataflow 
[13).  All  of  these  programming  models  require  the  same  ex¬ 
ecution  mechanisms:  communication,  synchronisation,  and 
translation.  Specialising  a  machine  for  a  particular  model  of 
computation  results  in  only  a  small  increase  in  performance. 

Concurrent  programming  it  not  difficult  if  suitable  abstrac¬ 
tions  we  used.  Programmers  should  use  the  natural  parti¬ 
tion  of  the  problem  and  not  be  concerned  with  placement. 
Synchronization  can  be  performed  by  allowing  the  data  flow 
of  the  program  to  sequence  the  required  operations.  As  this 
technology  matures,  we  expect  to  see  abstractions  for  con¬ 
currency  that  will  make  concurrent  programming  no  more 
difficult  than  sequential  programming. 

Many  challenging  problems  in  the  design  of  hardware  and 
software  for  concurrent  computers  remain.  A  major  research 
area  is  the  design  of  fault  tolerant  systems.  While  we  can 
construct  a  4K  node  machine  with  an  MTBF  of  2400  hours 
(4K  chips  at  100FITS),  future  machines  may  have  MTBFs 
of  only  a  few  hours  and  will  require  architectures  that  can 
survive  node  and  link  failures  without  lose  of  data. 

The  mechanisms  described  here  efficiently  execute  concur¬ 
rency  at  a  grain  size  of  3 its.  Many  numerical  programs, 
however,  have  potential  concurrency  at  the  level  of  single  op¬ 
erations.  Architectures  must  be  developed  that  can  exploit 
this  concurrency  without  incurring  the  overhead  of  message 
delivery  or  synchronisation. 

Another  critical  problem  is  the  development  of  (communica¬ 
tion.  processor,  and  memory)  resource  management  policies 
for  concurrent  operating  systems.  It  is  quite  easy  to  write  a 


program  frith  sufficient  concurrency  to  swamp  any  concur¬ 
rent  machine.  A  concurrent  operating  system  must  provide 
a  to  throttle  tech  such  massively  concurrent  applica¬ 

tions  to  match  the  concurrency  to  the  available  resources 

Concurrent  programming  systems  are  still  quite  primitive 
Abstractions  for  concurrency  that  express  common  patterns 
of  computation  while  hiding  the  details  of  implementation 
an  required  [11).  Compiler;  should  perform  optimizations 
that  expose  concurrency  in  programs  and  automate  the  place¬ 
ment  of  objects  onto  processing  nodes.  Concurrent  software 
technology  must  mature  for  them  powerful  machine!  to  see 
widespread  nee. 
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